E-Speech Therapy

Product design | Speech Visualization | Computer Graphics

Introduction

I undertook this project as my final year project for my undergrad course. Year-long research including ethnography, interviews, and field visits, contributed to the discovery of the answer to this question and the ideation of a potential technological solution. The project solution spans across fields like - digital signal processing, Animations, and gamification. The project is still in its development phase. However, I would like to share the research, design process, and the outcomes to date.

Team

Kritika Mittal
Yashvi Upadhyaya

Project Guide

Prof. Mayur Parulekar - Electronics
Prof. Anusha Vegesna - IT

Duration

October, 2018 - Present

Motivation

Being involved in projects related to accessible technologies of the hearing impaired, I came accross the statistics presented by the World health Organization. THe numbers were really surprising.

Over 5% of the world’s population – or 466 million people – has disabling hearing loss (432 million adults and 34 million children). It is estimated that by 2050 over 900 million people – or one in every ten people – will have disabling hearing loss.
The ratio of speech therapists to the deaf population was 1:200

This ignited in me thought "I think many hearing impaired cannot speak just because they do not have access to speech therapists"...To get answers to this, I planned to conduct a research study and was set on my venture to find a reasonable answer to this problem and ideate a solution to solve this.

Methodology

To proceed with the user research and field study I adopted the following methodology.

Step 1: Emapthize

"Empathy is the ability to fully understand, mirror, then share another person’s expressions, needs, and motivations."

Ethnographic Research

In order to get a deeper understanding of the problem, I planned to study disability as a whole and then jump deeper into the individual aspects of hearing impairement. I conducted enthonographis research which is a qualitative method where researchers observe and/or interact with a study's participants in their real-life environment. The following is the summary of the ethnographic research:

User Interviews

The ethnographic research helped me understand various types of impaired community, their lives, and the individuals(profession) that are related to them. With this, I could recognize the stakeholders of my research problem. They are:

Hearing Impaired Students

Speech Therapists and Teachers

Parents/Guardians of the Hearing Impaired Student

I enjoyed interviewing my stakeholders as it involved direct interaction with the individuals. For my interviews not only gave me the answers to the questions that I framed but also the in-person interaction helped me truly "Empathize" with what the individual thinks and feels.Here, I used my all-time fave, open-ended questions as I wanted my users to speak out whatever they wanted to rather than me directing them with my questions. Refer to this document with which I Interviewed the individuals.

Interviewing each of these stakeholders enabled me to get a big picture of what are the painpoints and what do these individauls expect to be solved. With this, I could create the three user personas, as follows:

Learnings from the Empathize Phase

About speech and Hearing Impairement
Difficulty that hearing impaired people face in the speech learning process over hearing people is the lack of a reception and feedback mechanism, which include listening to correct sound and pronounced sound. This hinders the hearing impaired person from evaluating the sound that he/she has produced. In such cases it is important for them to be able to visualize the speech. This includes looking at the movement of the lips and the tongue, identifying which sounds require the expulsion of air and which don’t, and most importantly be able to distinguish between sounds that have similar lip and tongue movements.

About their lives
The impaired are forced to live an isolated life along with the people who share a similar type of problem. This kind of isolation influences their life in various ways where they develop a fear of social involvement, feel a lack of opportunities. Well said by one of my professors: "This is just the initial stage of the lives of the hearing-impaired, where isolation can help them grow. But are we helping them prepare for the challenges of the world when they will earn themselves a living?"

From the Speech Therapists
he cause of speech impairment and the age of the patient are very important factors. The time it takes for speech therapy to show improvement, increases with age to a certain extent. Young children respond to speech therapy very fast as compared to older children or adults. This is due to the fact that young children (mostly under the age of 5) are fast learners. After a certain age, psychology of the patient also starts affecting the speech therapy process. Patients become more aware of their disability and due to social reasons may become reserved and hesitant to speak. Breaking the ice with these patients can take time and only after that does a patient become comfortable with their speech therapist and speech therapy starts showing results.

What is expected?
All the current systems give an objective evaluation which helps speech therapists to log the progress of a case. However, it does not give a method of correcting one of the mistakes made and help them improve their errors. Thus this software can only be used as an assistive device. The software which would independently enable the children and their families to access speech training and learning is something we really need to look for.

For detailed description refer to the detailed report: Qualitative Research Report

Step 3: Defining the Problem

"One learns to speak by listening to people speak and then trying to replicate those sounds they hear. This also includes a feedback mechanism which enables one to understand whether he/she has uttered the same sound or not. The process followed is, first the child hears a sound and observes the movement of the person’s mouth who produced that sound. Then he/she tries to replicate that movement and produce the same sound. Because they can hear what they sounded like, they can then try to correct themselves if what they said did not match what they had heard. However, for hearing impaired children, this process is difficult because while they can see the movement of the lips and tongue, they cannot hear the sound produced. Thus even if they try to replicate the movement of the mouth, generating the correct sound is difficult as the feedback mechanism is absent for them. Hearing loss can occur at any age but if it is present since birth, then it will be accompanied by muteness."

Step 3: Ideation

Having sufficient data to start with and a few guidelines from the users to what they required, I started building upon ideas. My teammate Kritika and I started sketching out a mindmap figuring out where various technological solutions could be implemented. A series of sketching and UX mapping helped us land to the idea of simulating the process of speech therapy on computers to teach children how to speak.

Mind Mapping

A mind map, to put it simply, is a tree that represents the central idea or topic and its subtopics as branches of that tree. Mind mapping is quite useful at organizing information that’s connected to one single topic and structuring it around that main idea

Concept Sketching

We discussed and defined how the system should work and to make it more clear to undertand and visualize, I sketched the following concept map.

Step 4: Prototyping

This is a pioneering project to develop for the Hearing-Impaired children of India. This project aims to teach the hearing-impaired children how to pronounce Devanagari sounds in order to develop speech. The project has 3 phases:

PHASE 1: Record Keeping /Data Collection Module
This will help the teachers and the speech therapists to keep track of all the sounds the student is practicing.
Comments and reports on the progress of the child can be maintained. These sounds will be private to the therapists and won't be shared with any other user.
Additionally, the sound samples collected via this module will be stored in the database which will be processed and used for training the module of phase 2.
Start using the module via this link : espeechtherapy.github.io

PHASE 2: Therapy Module
This is the actual therapy and self-learning module by which the hearing-impaired student can learn how to produce Devanagari sounds in a computerised way.

PHASE 3: Testing, Feedback and Improvisation
The usability and efficiency of the module will be tested on the users. Constant feedback and problems faced by the users would be worked up on and the system will be improved on this basis.

Phase 2 and Phase 3 are done in iterations.

Shhhh...

I cannot share with you the unpublished work of our project. Stay tuned to know more about the further developments in the project. To discuss more about the project feel free to contact me at yashviu@gmail.com

What am I learning?

"A problem well stated is a problem half solve" and a 'well-stated' problem is only possible with an extensive user study. It took me more than a year of interaction with the disabled society to truly empathize with them and the problems that they face. And I did realize that solutions are not to be found, they come from the users themselves. Be attentive when you listen to the users telling; 'Something like this would really help'.

Not all problems can be solved by technology. We started with the aim of making our system independent from speech therapists. However, throughout the process, we realized that the need to be a human involvement for a monitored functioning and progress of the therapy. Hence, we created a touchpoint for parents and speech therapists too, in the system.

Good things take time... Yes, we were bound by the college timeline which restricted the scope of this system. However, the work is still going on. We plan to proceed with the system and follow the ideate and validation phases in iteration until the therapists we collaborated with, give us approval.